System and method of overlaying images of different modalities

ABSTRACT

Systems and methods of overlaying a three-dimensional image of a tissue and an image of the tissue from an imaging transducer, having a field of view are disclosed. The method includes determining pixel dimensions of the field of view in the second co-ordinate space, scaling the three-dimensional image in the first co-ordinate space to match the pixel dimensions of the field of view in the second co-ordinate space, and displaying an overlaid image comprising the three-dimensional image and the field of view of the imaging transducer.

BACKGROUND

Medical imaging devices provide non-invasive methods to visualize theinternal structure of a patient. Such non-invasive visualization methodscan be helpful in treating patients for various ailments. For example,the early detection of cancer in a patient can be important in treatingthat patient. For most cancers, when detected at an early stage, thesurvival probability of the patient can increase.

There are many medical imaging methods available for visualizing theinternal structure of a patient, each with its own benefits and its ownlimitations. For example, non-invasive medical imaging techniques mayinclude MRI systems, three-dimensional (3D) X-ray computed tomography(CT) imagers, 3D optical coherence tomography imagers, or other 3Dmedical image of a tissue of a patient such as single proton emissioncomputed tomography, and position emission tomography.

Magnetic resonance imaging (MRI) uses magnetic fields to image tissue ofa patient placed inside a powerful uniform magnetic field of an MRIscanner. The MRI scanner can align the magnetic moments of protons inthe tissue (typically hydrogen protons of water molecules in the tissue)in the direction of the field, processing about the field at theirLarmor frequency. An excitation magnetic field (typically orthogonal tothe main magnetic field) near the Larmor frequency is applied to alterthe alignment of the protons in the tissue, typically flipping themagnetic moment of the protons in the main field. When the excitationfield is turned off, the protons emit a photon that can be detected andprocessed to form an MRI image of the tissue. Each image is a slice ofthe tissue of the patient, and numerous images or slices are createdthat clearly show all the features of the tissue of the patient. Theslices can be reconstructed into a single three-dimensional image,allowing complete visualization of the tissue of the patient scannedfrom all angles.

Other 3D imaging systems include tomosynthesis systems, which are X-raybased systems, and have recently been developed for use in breast cancerscreening. In contrast to typical mammography systems, the tomosynthesissystem acquires a series of x-ray projection images, each projectionimage obtained at a different angular displacement as the x-ray sourcetraverses along a path over the breast. Reconstructed tomosynthesisslices reduce or eliminate the problems caused by tissue overlap andstructure noise in single slice two-dimensional (2D) mammographyimaging. Digital breast tomosynthesis also offers the possibility ofreduced breast compression, improved diagnostic and screening accuracy,fewer recalls, and 3D lesion localization.

Ultrasound imaging is another non-invasive medical imaging technique,which uses sound waves, typically produced by piezoelectric transducers(or transducers) to image a tissue in a patient. The ultrasoundtransducer focuses the sound waves, typically producing an arc-shapedsound wave which travels into the body and is partially reflected fromthe layers between different tissues in the patient. The reflected soundwave is detected by the transducer and converted into electrical signalsthat can be processed by the ultrasound scanner to form an ultrasoundimage of the tissue. The ultrasound images formed may be 2D images or 3Dimages. The 3D ultrasound images may be formed by producing sound wavesat different viewing angles, multiplexing the reflected signals andprocessing the multiplexed signals to construct a three dimensional scanimage of a body object. In contrast with MRI and tomosynthesis systems,ultrasound systems typically produce a real-time stream of consecutiveimages.

SUMMARY

It is appreciated that each of the imaging techniques described above,including tomosynthesis imaging, MRI imaging and ultrasound imagingtechniques, have certain advantages and certain drawbacks. For example,ultrasound imaging tends to provide improved imaging of tendon structurein a patient over images of the same tendon structure provided by MRIimaging. Meanwhile, MRI imaging tends to provide superior soft-tissuecontrast resolution as compared to ultrasound images. For example,typical MRI imaging tends to allow individual structures such as a lung,liver, kidney, bowel, and gray and white matter to be distinguished inthe MRI images.

Another distinction includes the field of view (FOV) of the resultingimage. Ultrasound imaging generally provides a smaller field-of-view ascompared to MRI imaging, because the resolution of ultrasound imagestends to be restricted by the sound wave penetration through softtissues and bone. For example, ultrasound imaging has difficultypenetrating bone and thus typically only sees the outer surface of bonestructure and not what lies within.

Meanwhile, an advantage of ultrasound imaging as compared to MRI imagingis that ultrasound imaging provides real-time feedback. For example, anultrasound technician can position the ultrasound transducer directly ona patent in a first position and view the ultrasound image in real time.Subsequently, the technician can move the ultrasound transducer to asecond, perhaps more desirable position, to view the new ultrasoundimage, again in real time. This ability to adjust the position of thetransducer, while viewing the ultrasound image in real time, providesthe technician the ability adjust the ultrasound image until they aresatisfied with the displayed image. Real-time imaging can be helpfulduring interventional procedures. For example, during biopsy, ultrasoundimaging can help guide the movements of the biopsy tool in real-time asthe biopsy needle is inserted in the tissue.

It would be advantageous to combine the benefits of 3D medical imagingsuch as MRI imaging, single photon emission computed tomography,computed tomography, or positron emission tomography and ultrasoundimaging, by overlaying images of a tissue produced using multipleimaging techniques. Overlaying 3D images and ultrasound images producesa superimposed image that shows the features of the 3D image and thefeatures of the ultrasound image, and further takes advantage of thestrengths of both imaging systems. In addition, according to variousexamples, because the overlaid 3D and ultrasound images areco-registered as further described below, real-time changes in the FOVor zooming in or out of the ultrasound image, results in correspondingreal-time transformation of the superimposed image.

It is appreciated that current approaches are limited to using 3D and 2Dimages produced by an imaging manufacturer on a single platform having acommon video feed of the images. Typically, the single platform isequipped with a software upgrade that provides the image overlayingfunctionality. The software upgrade may be costly and is typically nottransferable to another machine to provide the same functionality.Instead, the systems and methods described herein for overlaying imagesare platform agnostic and do not require a common video feed. Forexample, systems and methods described herein are configured to receiveimages or video feed of different modalities supplied by independentimaging systems that may be made by different manufacturers. In variousexamples, the image processors described herein are configured todetermine the dimensions and orientation of the received images withoutany prior knowledge of the image features. These systems and methods canallow a workstation, such as a navigation system, to reformat anddisplay image data where that workstation is compatible with multiplecommercially available imaging systems. Further, the systems and methodsdescribed herein either store information regarding multiple platformsand transducers or can be calibrated to be used with multiple platformsat the clinical site.

According to one embodiment, a method overlaying a three-dimensionalimage of a tissue and an image of the tissue from an imaging transducer,having a field of view is disclosed. In one example, the methodcomprises co-registering a first co-ordinate space of thethree-dimensional image with a second co-ordinate space of the field ofview of the image from the imaging transducer, determining pixeldimensions of the field of view in the second co-ordinate space, scalingthe three-dimensional image in the first co-ordinate space to match thepixel dimensions of the field of view in the second co-ordinate space,transforming position and orientation of the three-dimensional image ofthe tissue in the first co-ordinate space to match position andorientation of the field of view in the second co-ordinate space, anddisplaying an overlaid image comprising the three-dimensional image andthe field of view of the imaging transducer.

In another example, determining pixel dimensions of the field of viewfurther comprises determining pixel coordinates of points of interest onan upper boundary of the field of view, and determining pixel width andheight based on geometry of the upper boundary of the field of view. Inthis example, the imaging transducer comprises a linear transducer, theupper boundary comprises a line segment, and the points of interestcomprise ends points and a midpoint of the line segment. In addition,determining the pixel width and height based on geometry furthercomprises determining the pixel width and height based on geometry ofthe line segment and aspect ratio of the image.

In one example, the imaging transducer comprises a curvilineartransducer, the upper boundary comprises an arc segment, and the pointsof interest comprise a midpoint and a radius of curvature. In thisexample, determining the pixel width and height based on geometryfurther comprises determining the pixel width and height based ongeometry of the arc segment and aspect ratio of the image.

In another example, the method further comprises detecting changes inimage depth of the field of view of the image and rescaling the field ofview based on a scaling factor to match an adjusted image depth of thefield of view. In addition, the method may further comprise receivingthe three-dimensional image of the tissue from a first imaging systemand receiving the image of the tissue from the imaging transducer from asecond imaging system, wherein the first imaging system is differentfrom the second imaging system.

According to another embodiment, an apparatus for overlaying athree-dimensional image of a tissue and an image of the tissue from animaging transducer, having a field of view is disclosed. In one example,the apparatus comprises a non-transitory computer readable mediumconfigured to store any of the three-dimensional image and the imagefrom the imaging transducer, a processor, coupled to the computerreadable medium and configured to co-register a first co-ordinate spaceof the three-dimensional image with a second co-ordinate space of thefield of view of the image from the imaging probe; determine pixeldimensions of the field of view in the second co-ordinate space, scalethe three-dimensional image in the first co-ordinate space to match thepixel dimensions of the field of view in the second co-ordinate space,and transform position and orientation of the three-dimensional image inthe first co-ordinate space to match the position and orientation of thefield of view in the second co-ordinate space, and a display configuredto display an overlaid image comprising the three-dimensional imageoverlaid and the field of view of the imaging transducer.

In one example, the processor is configured to determine pixelcoordinates of points of interest on an upper boundary of the field ofview and determine pixel width and height based on geometry of the ofthe upper boundary of the field of view. In another example, the imagingtransducer comprises a linear transducer, the upper boundary comprises aline segment, and the points of interest comprise ends points and amidpoint of the line segment. In one example, the imaging transducercomprises a curvilinear transducer, the upper boundary comprises an arcsegment, and the points of interest comprise a midpoint and a radius ofcurvature.

In one example, the processor is further configured to detect changes inimage depth of the field of view of the image and rescale the field ofview based on a scaling factor to match an adjusted image depth of thefield of view. In addition, the apparatus may further comprise an inputmodule configured to receive the three-dimensional image of the tissuefrom a first imaging system and configured to receive the image of thetissue from the imaging transducer from a second imaging system, whereinthe first imaging system is different from the second imaging system.

According to another aspect, a method overlaying an image of a tissuefrom an imaging system, having an imaging transducer with a field ofview, and a three-dimensional image of the tissue is disclosed. In oneexample, the method comprises co-registering a first co-ordinate spaceof the three-dimensional image with a second co-ordinate space of thefield of view of the image from the imaging transducer, receiving inputbounds of the field of view and depth markers identified in the image,determining pixel spacing based on depth associated with the depthmarkers in the image, and scaling the three-dimensional image in thefirst co-ordinate space to match the pixel spacing of the field of viewin the second co-ordinate space, transforming position and orientationof the three-dimensional image of the tissue in the first co-ordinatespace to match the position and orientation of the field of view in thesecond co-ordinate space, and displaying an overlaid image comprisingthe field of view of the imaging transducer and the three-dimensionalimage.

In one example, the method further comprises receiving depth associatedwith each of the depth markers in the field of view. In addition, themethod may further comprise receiving input identifying a freeze stateof the image from the transducer.

In another example, the method comprises determining co-ordinates ofpoints of interest defining center, upper and lower bounds of the fieldof view and determining pixel spacing based on the co-ordinates of thepoint of interest. In one example, the method further comprisesdetermining whether the field of view of the image is cropped.

In another example, the imaging transducer comprises a curvilineartransducer, and the method further comprises detecting an angle of thefield of view of the curvilinear transducer. In addition, the method mayfurther comprise determining a set of pixels configured to define thefield of view and masking out another set of pixels outside the definedthe set of pixels defining the field of view.

In another example, the method further comprises receiving thethree-dimensional image of the tissue from a first imaging system, andreceiving the image of the tissue from the imaging transducer from asecond imaging system, wherein the first imaging system is differentfrom the second imaging system.

According to another embodiment, an apparatus for overlaying an image ofa tissue from an imaging system, having an imaging transducer with afield of view, and a three-dimensional image of the tissue is disclosed.In one example the apparatus comprises a non-transitory computerreadable medium configured to store any of the three-dimensional imageand the image from the imaging transducer, a processor coupled to thecomputer readable medium and configured to co-register a firstco-ordinate space of the three-dimensional image with a secondco-ordinate space of the field of view of the image from the imagingtransducer, receive input identifying bounds of the field of view anddepth markers in the image, determine pixel spacing based on depthassociated with the depth markers in the image and scale thethree-dimensional image in the first co-ordinate space to match thepixel spacing of the field of view in the second co-ordinate space andtransform position and orientation of the three-dimensional image in thefirst co-ordinate space to match the position and orientation of thefield of view in the second co-ordinate space and a display configuredto display an overlaid image comprising the field of view of the imagingtransducer and the three-dimensional image.

In addition, the apparatus may further comprise an input moduleconfigured to receive the three-dimensional image of the tissue from afirst imaging system and configured to receive the image of the tissuefrom the imaging transducer from a second imaging system, wherein thefirst imaging system is different from the second imaging system.

According to another embodiment, a method of calibrating an imagingsystem for overlaying an image of a tissue from the imaging system and athree-dimensional image of the tissue, the imaging system having animaging transducer with a field of view, is disclosed. In one example,the method comprises providing for a user to select bounds of the fieldof view of the imaging transducer, providing for the user to selectdepth marker indicators in the image, cycling through the depth markerindicators to determine a specific depth marker for each depth, andcalculating coordinates of points of interest in the field of view ofthe image and calculating pixel spacing based on the coordinates of thepoints of interest and depth associated with the depth markers in theimage.

In another example, the method may further comprise providing for theuser to select an area identifying a freeze state of the image from thetransducer. In addition, the method may further comprise providing forthe user to select a background threshold, wherein the backgroundthreshold is used to filter an ultrasound signal overlapping with thedepth marker indicators. In one example, the method further comprisesdetermining whether the field of view of the image is cropped. Further,the imaging transducer comprises a curvilinear transducer, and themethod further comprises detecting an angle of the field of view of thecurvilinear transducer.

Still other aspects, embodiments, and advantages of these exemplaryaspects and embodiments, are discussed in detail below. Embodimentsdisclosed herein may be combined with other embodiments in any mannerconsistent with at least one of the principles disclosed herein, andreferences to “an embodiment,” “some embodiments,” “an alternateembodiment,” “various embodiments,” “one embodiment” or the like are notnecessarily mutually exclusive and are intended to indicate that aparticular feature, structure, or characteristic described may beincluded in at least one embodiment. The appearances of such termsherein are not necessarily all referring to the same embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of at least one embodiment are discussed below withreference to the accompanying figures, which are not intended to bedrawn to scale. The figures are included to provide illustration and afurther understanding of the various aspects and embodiments, and areincorporated in and constitute a part of this specification, but are notintended as a definition of the limits of the invention. In the figures,each identical or nearly identical component that is illustrated invarious figures is represented by a like numeral. For purposes ofclarity, not every component may be labeled in every figure. In thefigures:

FIG. 1 is a schematic diagram of a system used to implement methods ofoverlaying 3D images and the ultrasound images, according to variousembodiments;

FIG. 2 is a flow diagram of a method of overlaying 3D images and theultrasound images for a linear transducer, according to someembodiments;

FIG. 3 is a schematic diagram of a FOV of a linear transducer, accordingto some embodiments;

FIG. 4 is a flow diagram of a method of overlaying 3D images and theultrasound images for a curvilinear transducer, according to someembodiments;

FIG. 5 is a schematic diagram of a FOV of a curvilinear transducer,according to some embodiments;

FIG. 6 is a flow diagram of a method for dynamically detecting changesin image depth, according to some embodiments;

FIG. 7 is a flow diagram of an alternative method of overlaying an 3Dimage with an ultrasound image using symbol matching, according to someembodiments;

FIG. 8A is schematic diagram of one example of a platform levelcalibration interface for overlaying a 3D image and an ultrasound image,according to some embodiments;

FIG. 8B is schematic diagram of one example of portion of a platformlevel calibration interface for overlaying a 3D image and an ultrasoundimage, according to some embodiments;

FIG. 9A is schematic diagram of one example of a transducer levelcalibration interface for overlaying a 3D image and an ultrasound image,according to some embodiments;

FIG. 9B is schematic diagram of one example of a transducer levelcalibration interface for overlaying a 3D image and an ultrasound image,according to some embodiments;

FIG. 10 is schematic diagram of one example of angle detection used inthe transducer level calibration interface for overlaying a 3D image andan ultrasound image, according to some embodiments;

FIG. 11A is schematic diagram showing points of interest used in maskingmethods used for overlaying a 3D image and an ultrasound image for alinear transducer, according to some embodiments;

FIG. 11B is schematic diagram showing points of interest used in maskingmethods used for overlaying a 3D image and an ultrasound image for acurvilinear transducer, according to some embodiments; and

FIG. 11C is schematic diagram showing points of interest and circlesused in masking methods used for overlaying a 3D image and an ultrasoundimage for a curvilinear transducer, according to some embodiments.

DETAILED DESCRIPTION

According to various embodiments, systems and methods of overlaying afirst image produced by a first modality and a second image produced bya second modality are described below. In one example, the first imageproduced by a first modality includes a 3D image, such as an MRI imageacquired using an MRI imaging system, and the second image produced by asecond modality includes a 2D image, such as an ultrasound imageacquired using an ultrasound imaging system. It is appreciated thatwhile some systems and methods described below describe overlaying anMRI image and an ultrasound image, it is appreciated that other imagingmodalities can be used to overlay any imaging modality with that ofanother, for example, single photon emission computed tomography,computed tomography, positron emission tomography or 3D ultrasoundimaging.

As an illustrative example, one method of overlaying a 3D image and anultrasound image uses geometry of the ultrasound signal and ultrasoundtransducers to determine pixel dimensions in the ultrasound image. Thismethod uses the pixel dimensions to scale the 3D image to match thedimensions of the ultrasound image. Another method describes analternative method of overlaying the 3D image and the ultrasound imagethat uses symbol matching to identify depth markers and links the depthmarkers to a given depth and ultrasound settings. This method uses thedepth and the ultrasound settings to determine pixel spacing of theultrasound image, which is then used to scale the ultrasound image tomatch the scaling of the 3D image. It is appreciated that imagesproduced by either modality may be considered as a base image or anoverlaid image and either the base or the overlaid image may be scaledto match the other image.

FIG. 1 is a diagram of an embodiment of a system 100 used to implementthe methods of overlaying 3D images and the ultrasound images describedherein, according to various embodiments of the invention. The system100 includes a 3D imaging system 106, a tracking system 108, anultrasound imaging system 104, a communication system 110 and anavigation system 102. The 3D imaging system 106 and the tracking system108 are each connected to the navigation system 102 via thecommunication system 110 and the ultrasound image system 104 isconnected to the tracking system 108.

In summary according to various embodiments, the 3D imaging system 106obtains and stores one or more 2D image slices of a tissue of a patient.The stored 2D images can be compiled into the 3D image of the tissue,which can be displayed by the 3D imaging system 106. The 3D imagingsystem 106 transmits the 3D image to the navigation system 102, whichcan reformat the 3D image for display on the navigation system 102, asfurther described below. The ultrasound imaging system 104 obtains anultrasound image of a tissue of a patient using an ultrasound transducerand displays the image in real-time. The ultrasound imaging system 104can be configured for use with navigation system 102 by a calibrationprocess using the tracking system 108. Once the ultrasound imagingsystem 104 is calibrated as described below, the tracking system 108,using sensors, tracks the physical position, in which ultrasound imagingsystem 104 is imaging the tissue of the patient. Thus, through thecalibration process, the position and orientation of the ultrasoundimage plane can be determined relative to the sensors via the trackingsystem.

The navigation system 102 can use the position and orientation of theultrasound plane to determine the position and orientation of theultrasound transducer relative to the tracked position of the sensors.The navigation system 102 can then be configured to co-register andoverlay the ultrasound image and the 3D image using various embodimentsdescribed herein. According to various embodiments, the transducer maybe a linear or a curvilinear transducer. However, other transducershapes may be used as appreciated by one skilled in the art given thebenefit of this disclosure.

In more detail, and referring again to FIG. 1, according variousembodiments, the 3D imaging system 106 obtains a 3D image of the tissueof a patient and stores the image locally on 3D imaging system 106 or insome embodiments in a Picture Archiving Communications System (PACS).Typically, the image format of the 3D image is a DICOM format. However,skilled persons will understand that other image formats can be used.Once a tissue of a patient is imaged with the 3D imaging system 106, thestored image of the tissue can be reconstructed into a 3D image of thetissue and can be displayed by the 3D imaging system 106, or anotherworkstation. The 3D image, when displayed by an image viewer, such as aDICOM viewer, can be reformatted and repositioned to view the tissueimage at any plane and any slice position.

In one example, the 3D imaging system 106 transmits the 3D image to thenavigation system 102 via the communication system 110, for example viaa PACS network, where such 3D image can be stored and viewed. Thenavigation system 102 displays the 3D image obtained by the 3D imagingsystem. The 3D image can be reformatted for display on the navigationsystem 102. The 3D image can be reformatted and repositioned to view theimage at any plane and any slice position or orientation. In someembodiments, the navigation system 102 displays multiple frames orwindows on the same screen showing alternative views or orientations ofa 3D image slice.

According to various embodiments, the ultrasound imaging system 104obtains an ultrasound image of a tissue of a patient. The ultrasoundimage is typically obtained using an ultrasound transducer, which isused to image a portion of a tissue of a patient within a FOV of theultrasound transducer. The ultrasound imaging system 104 displays theobtained ultrasound image of the patient's anatomy in real-time as thepatient is being imaged. In some embodiments, the ultrasound image canadditionally or alternatively be stored on a non-transitory storagemedium, for reconstruction or playback at a later time.

In some embodiments, a real-time stream of consecutive two-dimensionalimages can be loaded into the navigation system 102 from an externalfeed from the ultrasound system 104. For example, these images can bestreamed using S-Video, HDMI, or other video streaming techniques. Insome embodiments, hardware used to receive such video streaming may beused to capture the streaming video. One example of video capturehardware includes a frame grabber card installed in the navigationsystem 102. However, it is appreciated that other methods of capturingultrasound images can be used.

The tracking system 108 may track the position of tracking sensorsconnected to the ultrasound imaging system 104 and provide thenavigation system 102 with data representing the co-ordinates of thetransmitters in a tracker co-ordinate space. The sensors may comprisetransmitter and may also function as receivers, for example when used inactive optical tracking systems. In some embodiments, the trackingsystem 108 may be an optical tracking system comprising an opticalcamera and optical sensors. However, skilled persons will understandthat any device or system capable of tracking the position of an objectin space can be used, for example an RF tracking system comprising an RFreceiver and RF transmitters.

The ultrasound imaging system 104 is configured for use with thenavigation system 102 by a calibration process using the tracking system108. In one embodiment, the calibration process is described in U.S.application publication no. 2011/0134113, the contents of which isincorporated herein by reference (hereinafter '113 application). Thesensors that are removably connected to the ultrasound transducer ofultrasound imaging system 104 may transmit position data to trackingsystem 108 in the tracker co-ordinate space, which in turn provides thisinformation to navigation system 102. For example, sensors may bepositioned on the transducer of the ultrasound imaging system 104 sothat the tracking system 108 can monitor the position and orientation ofthe ultrasound transducer and provide this information to navigationsystem 102 in the tracker co-ordinate space. The navigation system 102can use this tracked position to determine the position and orientationof the ultrasound transducer relative to the tracked position of thetransmitters.

In some embodiments, the configuration of the ultrasound imaging system104 occurs using a configuration tool, where its position andorientation can be additionally tracked by the tracking system 108. Oneexample of configuration of the ultrasound imaging system is describedin the '113 application. In the described example of configuration, theconfiguration tool contacts the transducer face of the ultrasoundtransducer of ultrasound imaging system 104 and tracking system 108transmits information representing the position and orientation of theconfiguration tool in the tracker co-ordinate space to the navigationsystem 102. The navigation system 102 can determine a configurationmatrix that can be used to determine the position and orientation of theFOV of the ultrasound transducer in the tracker co-ordinate space, basedon the tracked position of the transmitters connected to the ultrasoundtransducer.

In one example, a database having configuration data of a plurality ofbrands or models of various ultrasound transducers is used to pre-loadFOV dimensions of the ultrasound transducers into navigation system 102during configuration. In another example, the FOV dimensions of theultrasound transducers are obtained from the clinical site during acalibration procedure as further described below. For example, thevarious ultrasound transducers may include linear and/or curvilineartransducers.

Once the ultrasound imaging system 104 is calibrated with the navigationsystem 102, the tissue of the patient can be imaged with the ultrasoundimaging system 104. During ultrasound imaging, the tracking system 108monitors the position and orientation of the ultrasound transducer ofultrasound imaging system 104 and provides this information in thetracker co-ordinate space to the navigation system 102. Because theultrasound imaging system 104 has been configured for use with thenavigation system 102, the navigation system 102 may determine theposition and the orientation of the FOV of the ultrasound transducer ofthe ultrasound imaging system 104.

One example of configuration further includes the selection of alandmark within the 3D image, for example, using an interface permittinga user to select an anatomical target. In some embodiments, the landmarkcan be an internal tissue landmark, such as tendon, bone, veins orarteries, and in other embodiments, the landmark can be an externallandmark, such as a fiducial skin marker, such as a navel or nipple. Inthis embodiment, the same landmark selected in the 3D image is locatedwith the ultrasound transducer, and upon location, the coordinates ofthe representation of the target in the tracker co-ordinate space iscaptured.

In one example, the navigation system 102 uses the relative differencesbetween the coordinates of the target in the 3D image and theco-ordinates of the target in the tracker co-ordinate space to determinethe translational parameters required to align the two co-ordinatespaces. The plane orientation information acquired previously can becombined with the translation parameters to provide a complete 4×4transformation matrix capable of co-registering the two co-ordinatespaces. In at least one example, multiple landmarks may be selectedwithin the images, which can be used to capture multiple image planes.One example of co-registering the two co-ordinate space is described inthe '113 application. Any method of co-registering the co-ordinatespaces may be used, such as for example, a touch point fiducial markerregistration method.

The navigation system 102 can then use the transformation matrix toreformat the 3D image being displayed so that the slice of tissue beingdisplayed is in the same plane and in the same orientation as the FOV ofthe ultrasound transducer of ultrasound imaging system 104. Usingvarious methods described below, the slice of the 3D image beingdisplayed by navigation system 102 and the ultrasound image displayed byultrasound imaging system 104 can be combined by the navigation system102. The combined image allows the user to view both the 3D image andthe ultrasound image simultaneously, overlaid on the same display. Insome embodiments, the navigation system 102 can enhance certain aspectsof the superimposed ultrasound or the 3D images to increase the qualityof the resulting combined image. In addition, the navigation system 102can detect and correct for changes input by the user of the navigationsystem 102, including changes in FOV angle window, zoom and depth.

In the embodiments described above, the 3D imaging system 106, thetracking system 108 and the navigation system 102, together with theultrasound imaging system 104, are communicatively connected, via thecommunication system 110, in a stand-alone system. The tracking system108 is in communication with the navigation system 102 via thecommunication system 110 and tracks the physical position in whichultrasound imaging system 104 is imaging the tissue of the patient. Insome embodiments, the tracking system 108 can be connected directly tothe navigation system 102 via a direct communication system or awireless communication system. In some embodiments, the ultrasoundimaging system 104 can be communicatively connected via thecommunication system 110. In one example, the ultrasound imaging system104 is connected to the navigation system via a direct communicationsystem, such as, including but not limited to VGA, DVI, S-Video, orHDMI. The communication system 110 can also include a PACS network, alocal area network, wide area network, wireless network, internet,intranet, or other similar communication system.

In one embodiment, the 3D image of a patient can be accessed remotely bythe navigation system 102 via the communication system 110, and in otherembodiments can be stored on a server in communication with thenavigation system 102 via the communication system 110. In someembodiments, the navigation system 102 can access the ultrasound image,and in such embodiments the ultrasound imaging system 104 is furtherconnected to the communication system 110 and a copy of the ultrasoundimage obtained by the ultrasound imaging system 104 can be transmittedto the navigation system 102 via the communication system 110. In otherembodiments, the navigation system 102 can remotely access and copy theultrasound image via communication system 110, and in alternativeembodiments, a copy of the ultrasound image can be stored on a server incommunication with the navigation system 102 via the communicationsystem 110 and accessed remotely by navigation system 102.

The 3D image obtained by the 3D imaging system 106 can be transmitted tothe navigation system 102 at any point in time, for example immediatelyafter obtaining the 3D image, or on the request of the navigation system102. In alternative embodiments, the 3D image is transmitted to thenavigation system 102 by a transportable media device, such as a flashdrive, CD-ROM, diskette, or other such transportable media device.

According to various embodiments, the methods of overlaying the 3Dimage, such as an MRI image and the acquired ultrasound image aredescribed below. Method 200 describes a method of co-registering andoverlaying the ultrasound image over the 3D image where the ultrasoundtransducer is a linear transducer. The method 400 describes a method ofco-registering and overlaying the ultrasound image over the 3D imagewhere the ultrasound transducer is a curvilinear transducer. Method 600describes a method of dynamic depth detection used with methods 200 and400 which can be used to automatically detect changes in image depthdynamically throughout an imaging session. Method 700 describes analternative method of overlaying the 3D image and the ultrasound imagethat uses symbol matching to identify depth markers and links the depthmarkers to a given depth and ultrasound settings.

The methods described below relate to overlaying the 3D image and theultrasound image may be easily applied to using any 3D modality to beused with the ultrasound system, such as CT imaging, 3D medical imagingsuch as single photon emission computed tomography, computed tomography,or positron emission tomography or 3D ultrasound imaging. However, it isappreciated that some modification to the method may be necessarilybased on features and limitations of each of the above describedmodalities. For example, unlike the MRI modality, to overlay a 3D CTimage and an ultrasound image, the navigation system may limit thenumber and types of CT images available for overlaying.

Overlaying Using a Linear Ultrasound Transducer.

With reference to FIG. 2, a method 200 for co-registering and overlayingthe ultrasound image with the 3D image where the ultrasound transducerincludes a linear transducer is illustrated. The method 200 can beperformed on the navigation system 102, in real-time, as the ultrasoundimages (or video feed) are received from the ultrasound system.Alternatively, the method 200 can be performed at a later time, onimages received and stored on the navigation system 102. The method 200includes detecting and segmenting an upper boundary of a FOV of thelinear transducer, determining end-point and midpoints of the linesegment, determining pixel height and width of a pixel of theultrasound, scaling the co-registered 3D image and overlaying the 3Dimage on the ultrasound image. As described above, in one example, theFOV dimensions of the ultrasound transducers for a plurality of brandsor models of various ultrasound transducers may be stored and retrievedform a database. Alternatively, the FOV dimensions of the ultrasoundtransducers can be obtained from the clinical site during a calibrationprocedure further described below.

In step 202, the navigation system 102 detects and segments the upperboundary of the rectangular FOV in a single frame of the real-timeultrasound image stream. FIG. 3 shows one example of a detected singleframe 300 of an ultrasound image. The detected frame 300 includes pixels302 having a height (H) 308 and width (W) 310, the FOV of the linearultrasound transducer 304, and a line segment 306.

In step 204, the end points (labeled as endpt 1 and endpt 2 in FIG. 3)and the midpoint (labeled as midpt in FIG. 3) of the line segment 306matching the upper FOV border are determined in pixel co-ordinates (i,j). The pixel co-ordinates (i, j) represent the pixel indices within theimage frame. For example, the pixel in the upper left corner of theimage frame is associated with a first index value (1,1). In otherexamples, the pixel co-coordinates can take on continuous values. Pixelcoordinates having continuous values may be used with a curvilineartransducer, as further discussed below.

In some embodiments, the detection of the line segment 306 may beassisted by having the user select a point in the image just above theupper FOV boundary. For example, the user can select the point using apointing device (e.g. a mouse). Using the selected point, the navigationsystem 102 can perform a downward directed search to identify the firsthorizontal line segment in the image.

In step 206, the navigation system 102 determines the pixel width (W) ofthe pixels in the single image frame 300. In one example, the pixelwidth is determined as a known width of the linear ultrasound transducerin mm (Wt) divided by the length of the corresponding line segment inpixels (L): W=Wt/L. The length of corresponding line segment (L) may bedetermined by determining a difference between endpt 1 and endpt 2.

In step 208, the navigation system 102 determines the ultrasound pixelheight (H). In one example, the height is determined as the width (W)divided by the aspect ratio (AR) of the image frame: H=W/AR. ARrepresents a proportional relationship between the width and the heightof the pixels within the ultrasound image. Typically, the AR for a giveninput video format is known and fixed. In the embodiments describedherein, it can be advantageous if the pixels in the two-dimensionalimage stream have a fixed AR that is equal to the pixel width divided bythe pixel height. While the actual pixel size in millimeters (mm) maychange if the ultrasound image is zoomed in or out, the AR remains thesame.

In step 210, the navigation system 102 uses the ultrasound pixel size(W×H) to scale the co-registered 3D image so as to have the same visualscale as the ultrasound image. In one example, the 3D image is selectedthat matches the current tracked orientation of the ultrasound FOV,which can be detected using the tracking system 108. The scaling iscompleted, for example, by matching the pixel dimension of the 3D imageto the pixel dimensions of the ultrasound image. The scaling of the 3Dimage may allow the overlay of the 3D image and the ultrasound imagewithout any inappropriate distortion or “stretch” of the MR image.

In step 212, the navigation system 102 overlays the co-registered 3Dimage and the ultrasound image stream by translating and rotating the 3Dimage. In one example, after the 3D image is scaled, the 3D image can betranslated so that the known location of the center of the transducerface in the 3D image coincides with the mid point of the segmented upperFOV boundary in the ultrasound image. Once the 3D image is translated,the 3D image may be rotated so that the known direction vector of thetransducer axis, within the 3D image, points vertically downward in theultrasound image, and the known direction of the “lateral” transduceraxis points horizontally (in the appropriate direction, either Left orRight) in the ultrasound image.

In some embodiments, the navigation system 102 can repeat the method200, every pre-determined time interval, to detect changes in zoom. Achange in the zoom level may change the length of the upper FOV boundaryand the pixel size in millimeters (mm). By continuously performing themethod 200 every pre-determined time interval, the 3D image can beoverlaid accurately even if the zoom level is changed. Thepre-determined time interval can be selected by the user of thenavigation system. For example, selecting a shorter time interval canprovide for faster response time to changes in the zoom level, but mayalso necessitate more processing power.

Overlaying Using a Curvilinear Ultrasound Transducer

With reference to FIG. 4, one example of a method 400 for co-registeringand overlaying the ultrasound image and the 3D image, where theultrasound transducer has a curvilinear transducer, is illustrated. Themethod includes detecting and segmenting an upper boundary of a FOV ofthe curvilinear transducer, determining pixel co-coordinates of themidpoint of the upper FOV boundary, determining ultrasound pixel size,scaling the co-registered 3D image and overlaying the 3D image and theultrasound image.

In step 402, the navigation system 102 detects and segments the upperboundary of the sector shaped FOV in a single frame of the real-timetwo-dimensional image stream. FIG. 5 shows one example of a detectedimage frame 500 of an ultrasound image using a curvilinear transducer.The detected frame 500 includes, pixel co-ordinates (i,j) 502, the FOVof the curvilinear ultrasound transducer 504, an arc segment 506representing the upper curved boundary of the FOV having a midpoint, andan ellipse to fit the curve line segment 510. Similar to method 200,pixel co-ordinates are determined for the frame of the ultrasound image.In one example, a series of pixel co-ordinates 508 are determined thatare estimated to lie along the upper curved boundary of the FOV. Thesepixel co-ordinates may take on continuous values as the curve may notpass through the exact center of each pixel containing the curve.

In step 404, the navigation system 102 determines the pixel co-ordinatesof the midpoint 510 on this curve. In one example, the midpoint may bethe lowest point on the curve arc segment 506.

In step 406, the ellipse 510 with radius of curvature (ROC) parameters,ROCx, and ROCy, is fit to the curve segment 506. By fitting the ellipse510 to the curve segment 506, the ultrasound pixel size can bedetermined. The ellipse can be represented by expression:

(x−x _(o))²/ROCx ²+(y−y _(o))²/ROCy ²=1  Equation (1)

where x is the horizontal axis within the ultrasound image pixelco-ordinate system, y is the vertical axis within the ultrasound imagepixel co-ordinate system, and x_(o) and y_(o) represent the center pointof the ellipse. In the example shown, ROCx and ROCy represent the“radius” of the ellipse along its semimajor and semiminor axes inultrasound pixels (rather than mm).

According to various examples, where AR equals 1, ROCx equals ROCy, acircle, rather than an ellipse, is used to determine the pixel size.Where the AR of the ultrasound image is not equal to 1, an ellipse canused to determine pixel size. In this example, the circular arc becomesstretched along one dimension when considered in pixel co-ordinates.

The center point of the ellipse (x_(o), y_(o)) can be determined inrelation to the determined midpoint: with x_(o) equal to the midpoint ofthe segmented curved upper boundary of the FOV; and y_(o) equal to the yco-ordinate of the midpoint (for example, the lowest point) on the upperboundary of the curved FOV plus ROCy. Thus, in this example, Equation 1contains only two degrees of freedom for fitting to the upper FOVboundary, ROCx and ROCy.

In step 408, the ultrasound pixel size can now be determined bydetermining width and height of the pixel in relation to the knownradius of curvature of the curvilinear transducer. The width (W) andheight (H) can be expressed as follows:

W=ROCmm/ROCx  Equation (2)

H=ROCmm/ROCy  Equation (3)

where ROCmm is the known ROC of the transducer in mm.

In embodiments where the AR may not by known in advance, the AR can becomputed as AR=W/H. In embodiments where the AR is known in advance, thefitting of the ellipse can be constrained to a single degree of freedom,which may improve the accuracy of the fit of the ellipse to the upperboundary of the FOV. In this embodiment, the known AR can be used incombination with Equation 2 and Equation 3 to express ROCy in terms ofROCx and can be represented by dividing Equation (2) by Equation (2):

AR=ROCy/ROCx  Equation (4)

It then follows that ROCy=AR×ROCx and substituting into Equation (1)results in an ellipse with a single degree of freedom, ROCx, expressedby:

(x−x _(o))²/ROCx ²+(y−y _(o))²/ROCy ²=1  Equation (5)

In step 410, the ultrasound pixel size (W×H) can be used to scale theco-registered 3D image, so as to have the same visual scale as theultrasound image. In one example, the 3D image is selected that matchesthe current tracked orientation of the ultrasound FOV, which can bedetected using the tracking system 108. The scaling is completed, forexample, by matching the pixel dimension of the 3D image to the pixeldimensions of the ultrasound image. The scaling of the 3D image mayallow the overlay of the 3D image over the ultrasound image without anyinappropriate distortion or “stretch” of the MR image.

In step 412, the navigation system 102 overlays the co-registered 3Dimage and the ultrasound image stream by translating and rotating the 3Dimage. In one example, after the 3D image is scaled, the 3D image can betranslated so that the known location of the center of the transducerface in the 3D image coincides with the mid point (midpt) of thesegmented upper FOV boundary in the ultrasound image. Once the 3D imageis translated, the 3D image may be rotated so that the known directionvector of the transducer axis, within the 3D image, points verticallydownward in the ultrasound image, and the known direction of the“lateral” transducer axis points horizontally (in the appropriatedirection, either Left or Right) in the ultrasound image.

In some embodiments, the navigation system 102 can repeat the method400, every pre-determined time interval, to detect changes in zoom. Achange in zoom level may change the length of the upper FOV boundary andthe pixel size in millimeters (mm). By continuously performing themethod 400 every pre-determined time interval, the 3D image can beoverlaid accurately even if zoom level is changed. The pre-determinedtime interval can be selected by the user of the navigation system. Forexample, selecting a shorter time interval can provide for fasterresponse time to changes in zoom level, but may also necessitate moreprocessing power.

Dynamic Depth Detection

According to various embodiments, for some ultrasound systems, to insurethat the displayed ultrasound image occupies as much area on theultrasound display as possible, the ultrasound image displayautomatically re-scales its pixel dimensions when a user manually adjustthe image depth. This functionality may make it challenging todynamically reformat a 3D image dataset to match a real-timetwo-dimensional ultrasound stream, as the ultrasound image size andpixel dimensions may change part-way through an imaging session,particularly if the operator adjusts the image depth of the ultrasoundsystem.

For a subset of ultrasound systems, the default configuration is suchthat the number of pixels in the vertical dimension of the displayedultrasound image is constant regardless of image depth. For this subsetof ultrasound systems, the width in pixels of the displayed imagedynamically changes when the operator adjusts the image depth. In thesecases, methods described below can be used to automatically detectchanges in image depth dynamically, even midway through an imagingsession, and to rescale the image accordingly.

With reference to FIG. 6, one example of a method 600 for dynamicallydetecting changes in image depth of an ultrasound image is illustrated.In step 602, once every N frames of the ultrasound video stream (whereN>1), the transducer dimensions are automatically determined using themethods described above. The transducer dimensions may include the widthin pixels, for a linear transducer, or the radius of curvature (ROC) inpixels, for a curvilinear transducer.

In step 604, the navigation system 102 compares the dimensionsdetermined in step 602 to the determined dimensions for a previousframe, calculated N frames previously. If the dimensions are unchanged(step 606), then the depth of the image has not changed. In thisexample, no rescaling of the reformatted 3D image display is needed andthe method 600 ends.

In step 606, if the dimensions calculation result differs from theprevious result (calculated N frames previously), then the operator hasmanually adjusted the depth of the ultrasound image. In this example,the reformatted 3D image display needs to be re-scaled accordingly andthe method 600 proceeds to step 608.

In step 608, the navigation system 102 determines a scaling factor,which may allow the 3D image to be re-scaled to match the adjusted depthof the ultrasound image. The scaling factor for a linear transducer canbe determined as follows:

K=Width₂/Width₁  Equation (6)

where Width1 is the new width calculation (in pixels) and Width2 is theprevious width calculation (in pixels). For a curvilinear transducer,the rescaling factor can be determined as follows:

K=ROC₂/ROC₁  Equation (7)

where ROC₁, is the new ROC calculation (in pixels) and ROC₂ is theprevious ROC calculation (in pixels). In cases where both ROCy and ROCxare defined, either one can be used for purposes of determining thescale factor K.

In step 610, the scaling factor K can be used to determine the new imagedepth, where depth_(new)=K×depth_(old). The reformatted 3D image displayand associated annotations can then be updated to reflect the adjustmentin the currently displayed ultrasound image depth.

Overlaying the Ultrasound Image Using Depth Detection

Another method 700 of ultrasound image overlaying the 3D image and theultrasound image is described with reference to FIG. 7. The ultrasoundoverlay depth detection 700 uses symbol matching, where a depth markeris identified on the ultrasound image's annotations and linked to agiven depth and settings. The key advantage of the method 700 over theprevious method described above with reference to FIGS. 2-6 is that theultrasound signal may not be analyzed. Because the depth detection isindependent of the ultrasound signal, the method 700 allows the user toadjust any zoom or FOV settings on the ultrasound system.

As described in more detail below, the method 700 of depth detectionincludes a two-step calibration procedure. The first step in thecalibration procedure includes an ultrasound system (or platform level)calibration that is performed once per ultrasound system. The secondstep includes a transducer specific calibration that is performed onceper transducer on any given ultrasound system. Both calibrationprocedure steps are intended to be service level procedures. It isappreciated that the method 700 and the calculations performed part ofthe method can be used for both linear and curvilinear ultrasoundtransducers. The information received and determined as a result of boththe platform level calibration and the transducer calibration, such asthe bounds of the FOV, the depth markers in the image, and the pixelspacing may be received by the navigation system 102. The informationreceived may be used to determine pixel spacing of the ultrasound image.

It is appreciated that any image processing methods or techniques may beused in performing the various calibration steps to detect and identifyvarious image features. For example, whether or not a given pixelcontains part of the ultrasound signal or contains a particularannotation can be determined by comparing the pixel to the backgroundthreshold (or pixel intensity threshold). In addition, methods ortechniques of detecting text in individual image frames can be used todetermine text characters in the ultrasound images.

In step 702, the ultrasound system (platform level) calibration isperformed, which includes receiving identified the bounds of the FOV inthe ultrasound image and defined depth markers within the ultrasoundimage. The steps of performing platform level calibrations 710-714 aredescribed with reference to FIGS. 8A-8B.

In step 704, the transducer specific calibration is performed, whichincludes detection of the center, upper and lower coordinates of theFOV, determining ultrasound pixel spacing and for each possible depthfor the ultrasound transducer, specifying the current depth in thecalibration interface, and confirming that the depth marker and theultrasound FOV are identified correctly. The steps of performingtransducer level 716-722 calibrations are described with reference toFIGS. 9A-9B.

In step 706, the ultrasound pixel spacing can be used to scale theco-registered 3D image so as to have the same visual scale as theultrasound image. In one example, the 3D image is selected that matchesthe current tracked orientation of the ultrasound FOV, which can bedetected using the tracking system 108. The scaling is completed, forexample, by matching the pixel dimension of the 3D image to the pixeldimensions of the ultrasound image. The scaling of the 3D image mayallow the overlay of the 3D image and the ultrasound image without anyinappropriate distortion or “stretch” of the 3D image.

In step 708, the co-registered 3D image and the ultrasound image streamcan then be overlaid by translating and rotating the 3D image. In oneexample, after the 3D image is scaled, the 3D image can be translated sothat the known location of the center of the transducer face in the 3Dimage coincides with the points of interest (A and B), and the detectedcenter, upper and lower coordinates. Once the 3D image is translated,the 3D image may be rotated so that the known direction vector of thetransducer axis, within the 3D image, points vertically downward in theultrasound image, and the known direction of the “lateral” transduceraxis points horizontally (in the appropriate direction, either Left orRight) in the ultrasound image. In some embodiments, the ultrasoundimage stream can be overlaid on the 3D image.

Ultrasound System Specific Calibration

The steps of performing ultrasound system calibration (step 702) aredescribed with reference to FIGS. 8A-8B, which shows one example of acalibration interface 800 on the navigation system. In one example, thecalibration performed prior to real-time ultrasound signal detection andmay only be performed once for a given ultrasound system. Referring toFIG. 8A, the user can assign the name of the ultrasound system to becalibrated 802. In some examples, the user can adjust other settings fora given ultrasound system. For example, the user can adjust analogsettings of the video signal if necessary to adjust for backgroundnoise. Adjusting the analog settings may ensure that the correct AR ofthe video signal is maintained.

In step 710, for each ultrasound FOV, according to one example, the userdraws a rectangle 804 to identify the bounds within which the FOV isfully contained. Selecting the FOV assists in limiting the imageprocessing methods, described above with reference to method 700, toprocesses those locations of the ultrasound image that contain therelevant ultrasound signal. When selecting the FOV, the user may beinstructed to ensure that a ruler annotation associated with the depthannotations is not included in the rectangle 804.

In step 712, once the ultrasound FOV is identified, the user draws arectangle 806 to identify the bounds within which the depth markerindicators are fully contained. The identified depth marker can take theform of numerical annotations and may also include a ruler-associateddepth marker (shown in FIG. 8A). In other examples, the identified depthmarkers may take the form of simple text (not shown), used in additionor instead of the numerical annotations. The user may be instructed toavoid selecting the ruler as part of the rectangle 806. Additionally,the depth marker may change position depending on the current depth.Identification of depth markers is described further below.

After the depth indicators are selected, the user can be instructed tocycle through all possible depths indicators to ensure that a numericindicator for each depth is contained within the region. In a portion ofthe user interface shown in FIG. 8B, the user is shown the numericindicator for the depth of 20 is not contained within the region 806(shown on the right). The user can then be instructed to adjust therectangle 806 to contain the depth indicator of “20” (shown on theleft).

Referring again to FIG. 8A, the user can set a background threshold 808for the depth marker. It is appreciated that in some examples, theultrasound systems allow the ultrasound signal to pass through the depthmarker region of interest (as shown in FIG. 8A). Setting the backgroundthreshold 808 assists in filtering the ultrasound signal in cases wherethe depth marker and ultrasound signal overlap. In these examples theuser can select the background threshold 808 at which to ignore (orfilter) any background signal, including the ultrasound signal, behindthe depth markers.

It is appreciated that some ultrasound systems provide a visualindication when the ultrasound signal has been frozen (or paused). Inthese examples, in step 714, the user may identify an on-screen symbol810 that indicates the ultrasound video has been frozen, by selecting arectangle that contains this symbol. The selection of the symbol 810 inthe frozen state is compared against the selection in the live video (orstreaming) stage to determine whether the ultrasound signal has beenfrozen. In one example, the visual indication of the freeze marker mayinclude a general matrix representation of the pixel data identified inthe freeze marker region of interest, which is compared to the freezemarker region of interest from the live image. It is appreciated thatthe frozen selection is optional and may not be available on allultrasound systems. In the examples where no region is defined, freezestate detection is not performed for that ultrasound system and the usermay be instructed to skip the freeze state identification.

As noted above, different types of depth markers can be used bydifferent ultrasound systems to indicate ultrasound signal depth.Stationary depth markers explicitly display the current depth, whileruler-associated depth markers show a numeric value that is equal to orless than the current depth and use tic marks to show incrementalincreases. For ruler-associated depth markers, while a tic mark may bereused for multiple depths, the location of this symbol can be differentfor each depth. For example, “10” depth marker can be used for a depthof 10 cm and 11 cm. By storing the pixel location and symbol for eachdepth, unique markers for any given depth of an ultrasound transducercan be identified.

To find the unique depth marker for each depth, the ultrasound systemcan search the identified depth marker region of interestsystematically. For example, the ultrasound system can search the depthmarker region from left to right, then bottom to top. Such systematicsearching ensures finding the depth marker in the region of interest,which may be the most meaningful for accurately overlaying the FOV.

Once a pixel is found that is above the calibrated background threshold808, the left, right, top and bottom bounds of the marker can be found.A matrix can then be created and populated as a binary representation ofthe pixel data. In one example, the matrix can include a value of “0” toindicate pixel intensity less than the threshold, while a value of “1”indicates pixel intensity greater than or equal to the threshold. It ispossible that in cases where the depth marker overlaps the ultrasoundsignal, the user could greatly increase the gain of the ultrasoundsignal, causing the pixels to increase over the calibrated threshold808. Therefore, the threshold should be set to the highest possiblevalue, while still maintaining the ability to identify unique depthmarkers. In one example, the depth markers overlapping the ultrasoundsignal have had the associated annotations set to white, allowing for athreshold value of 255 and greatly reducing the possibility of errorsdue to overlapping ultrasound signal.

Transducer Specific Calibration

Once the particular ultrasound system is calibrated, the individualtransducers can be calibrated for use with the ultrasound system (step704) via the navigation system. The steps of performing transducercalibration are described with reference to FIGS. 9A-9B. Referring toFIG. 9A, within a user interface 900 presented to the user on theultrasound system, the user can select the type of transducer forcalibration. The transducer type can be chosen from a list of predefinedtransducers, specific to the ultrasound platform in use. The userinterface may also provide a description of the transducers that includeinformation such as radius of curvature (ROC) and the maximum FOV angle.In response to the transducer type selection, an ultrasound image isloaded into the system. The navigation system then detects points ofinterest 904 in the FOV 906, including the center, and the upper andlower coordinates, which are used to overlay the center, as well as theupper and lower bounds of the FOV in the ultrasound image 902 (step716). The points of interest 904 are also used in determining the pixelspacing, and in determining the center of the transducer in the image asdescribed further below. FIG. 9B shows one example of the ultrasoundimage 902 having points of interest 904 in the FOV 906 for a curvilineartransducer. The points of interest 904 include points A and point Bhaving the coordinates of interest including the x-coordinate, whichbisects the ultrasound signal, representing the y-coordinate of theupper boundary of the ultrasound FOV 906, and representing they-coordinate of the lowest point in the image where ultrasound signal isdisplayed.

The coordinates of the points of interest are found for each depth inthe transducer specific calibration as described below, which areexpressed in pixels. Although FIG. 9B shows the FOV 906 for acurvilinear transducer, it is appreciated that the transducercalibration steps can be used for a linear transducer. Determining thex-coordinate of the center of the image eliminates the need to find thecoordinate in the processing of each image. In addition, determining thex-coordinate ensures that the search for matching depth symbols startsat an appropriate location in the ultrasound image.

The center of the ultrasound FOV signal is first identified by searchingdown from the top of the ultrasound region of interest identified in thesystem level calibration described above. When the first non-zero pixelis reached, the edges of the ultrasound signal are then located bysearching left and right. The middle point between the left and rightedges is determined to be the center x-coordinate of the ultrasoundsignal. Given this center x-coordinate, the upper and lower bounds ofthe ultrasound signal are identified by searching along the centercolumn for the first non-zero pixel. The upper boundary y-coordinate isdefined as the first non-zero pixel when searching down from the top andthe lower boundary is defined as the first non-zero pixel when searchingup from the bottom.

Based on the x and y-coordinates, the center line and the upper andlower boundary lines representing the boundaries of the FOV can then beoverlaid over the ultrasound image. In addition to the center and upperand lower bounds, the ultrasound image can be overlaid with the boundsof the identified depth markers determined in the ultrasound systemlevel calibration described above.

The pixel spacing can be calculated as a function of the y-coordinate ofthe lower boundary and the upper boundary, as well as the current depthof the ultrasound image (step 718). The interface 900 can then displaythe pixel spacing 908. In one example, the current depth of theultrasound image is entered by the user within the ultrasound system.The pixel spacing can be expressed as follows:

pixelSpacing=(lowerBoundary_(y)−upperBoundary_(y))/depth  Equation (8)

According to one example, the pixel spacing in the x- and y-directionsis assumed to be equal, based on typical ultrasound systems. However, inother examples, the pixel spacing in the x and y directions can bedifferent. As noted above, Equation 8 can be used for both linear andcurvilinear ultrasound transducers.

To further calibrate the imaging transducer for correct depth markers,the user, via the interface 900, may be instructed to cycle through eachpossible depth for the ultrasound transducer and specify the currentdepth in the calibration interface 900. The calibrated depths may bedisplayed in the calibrated depth box 914. For each depth, the user canbe instructed to confirm that the depth marker and the ultrasound FOVare identified correctly. If the depth is not identified correctly, theuser can update the depth number box 910 to match the current depth ofthe ultrasound transducer.

In some embodiments, the navigation system may determine that theultrasound signal is cropped at the bottom of the ultrasound image (step720). In these embodiments, the cropped signal can lead to thecalculation of an incorrect pixel distance and ultimately, incorrectpixel spacing. To compensate for cropped signal, the user can draw arectangle 912 over a ruler displayed in the ultrasound image. The regionidentified as containing the ruler is searched in a similar manner tothe depth marker detection in order to find the lowest ruler marker. They-coordinate of the center of this marker can also be determined. Ifthis coordinate is found to be outside the upper and lower bounds of theidentified ultrasound region of interest, the coordinate identified asthe lowest point of the ultrasound signal is used. If the ruler markeris within the ultrasound region of interest, the base point in the imageis determined as the lower of the ruler coordinate and the lowestultrasound signal coordinate. When every depth setting has beencalibrated, each calibrated depth is stored for the selected transducercalibration and the transducer calibration process is completed.

Depth Detection while Tracking

The depth may be dynamically detected and updated while the navigationsystem 102 is in use with the tracking system 108. The depth markerregion of interest 806 can be searched to find the lowest depth marker916. The lowest depth marker 916 can be compared with the calibrateddepth markers associated with the transducer in use, which arepreviously stored in the navigation system 102. If a match is found, thetransducer's FOV depth is updated to reflect the updated depth, thecorresponding calibration settings are applied to the overlaidultrasound image. The overlay mask, further described below, can also beupdated to reflect the updated FOV settings. If no match to the depthmarker is found, the calibration settings for the transducer's FOV arenot changed.

Field of View Angle Detection

As described above, the calibration steps may be performed for acurvilinear or linear transducer. However, for a curvilinear transducer,the ultrasound system performs the additional steps of angle detectionfor the given ultrasound FOV. The detected FOV angles are used foroverlay masking methods described below. To find the left and right FOVangles, a line from the origin to the lower curve of the ultrasound FOVis defined, starting at the maximum FOV angle defined for thetransducer.

FIG. 10 shows outer lines 1002 which indicate the maximum FOV angle.These outer lines 1002 are scanned and if any non-zero pixels are foundinside the calibrated ultrasound region of interest, the angle of theline may be set as the FOV angle. If there are no non-zero pixels on theline, the angle may be incremented by a predetermined amount, and thenew line is scanned. For example, the predetermined incremented amountmay be 0.25 degrees, and may be configured to balance the accuracy ofthe resulting angle with the computational cost of using smallerincrements. However, other predetermined incremented amounts can beused. The FOV angles may be found by sweeping the line from left toright and vice versa. If there is a change in the FOV angles, thetransducer's properties and the overlay mask are updated.

It is appreciated that in order for the ultrasound system to identifythe correct FOV angle, the ultrasound system assumes a non-zero valueset on the outermost edge of the ultrasound signal. However, in someexamples, if the ultrasound signal gain is set to a low value, there maynot be a non-zero value identified by the ultrasound system, resultingin detection of an incorrect angle. An annotation graphic may beincluded in the interface 900 that allows for the user to lock thecurrent FOV angle, which may be useful in examples where the signal gainis set to a low value.

Masking the Ultrasound Overlay Image

In the embodiments described above, when applying the ultrasound imageas an overlay over 3D images, further image processing may be used toremove any unwanted information from the ultrasound image or applyvisual effects to the ultrasound image. For example, to remove anyinformation other than the ultrasound signal, the ultrasound system canmask the ultrasound overlay image. Masking can be applied by firstcreating a binary image of the ultrasound image with the same number ofpixels along the x and y axes and assigning a “1” to any pixel to bedisplayed and a “0” to any pixel to be removed. The binary image isdetermined by finding a collection of points in the image that candescribe the position and shape of the ultrasound FOV. Using thecollection of points, the ultrasound system can determine which pixelsare within the FOV of the ultrasound image and which ones remain outsidethe FOV boundaries. The mask may be applied to the ultrasound imagethough a fragment shader.

Linear Ultrasound Transducers

Because of the geometries of the FOV for curvilinear and lineartransducers, the masking process differs for each transducer. Todetermine the binary image for linear ultrasound transducers, theultrasound system uses the y-coordinates of the upper and lowerboundaries of the ultrasound FOV and the x-coordinates of the left andright boundaries of the FOV, as determined above in calibrating thetransducer.

As described above, the lower boundary remains constant, while the upperboundary and left boundary are found in searching for non-zero pixels.The right boundary of the image can be determined as:

rightBoundary_(x)=center_(x)+(center_(x)−leftBoundary_(x))  Equation (9)

FIG. 11A shows the four boundaries of the FOV of a linear transducer.The four points on a linear transducer ultrasound image needed todescribe the shape of the FOV are shown, with point A representing theTop Boundary, point B representing the Left Boundary, point Crepresenting the Right Boundary, and point D representing the BottomBoundary. The pixels within the four boundaries are set to “1,” allother pixels are set to “0.” As the result, any pixel that is outsidethese boundaries is masked out.

Curvilinear Ultrasound Transducers

To mask a curvilinear image, six points in the ultrasound image areneeded to be identified, including the base of the upper curve of theFOV, the base of the lower curve of the FOV, the top points of the leftand right field lines (which also correspond to points on the uppercurve) and the bottom points of the left and right field line (whichcorrespond to points on the lower curve).

FIG. 11B shows the six points including A=Top of Left Field Line, B=BasePoint of Upper Curve, C=Top of Right Field Line, D=Bottom of Left FieldLine, E=Base Point of Lower Curve, F=Bottom of Right Field Line,G=Origin of the FOV Lines, Center of Upper Curve.

Through the transducer specific calibration, described above, the uppercurve base point and the current depth and the pixel spacing have beendetermined. The y-coordinate of the base of the lower curve, can bedetermined by:

lowerCurve_(y)=upperCurve_(y)+(depth×pixelSpacing)  Equation (10)

In addition, using the FOV angle of the ultrasound signal, as describedwith reference to FIG. 10, the slope of the left and right field lines(from points A to D and C to F) can be calculated. To find the top ofthe left and right FOV lines (ABC), the length of these lines is set tothe radius of curvature of the upper FOV curve and the origin of theselines to the center point of the upper FOV curve. To find the bottompoints of these lines (DEF), the length of these lines is increased bythe current depth, and multiplied by the pixel spacing. Using the bottomfield line points (DEF) and the base point of the lower curve (ABC), thecircle that fits the lower curve of the FOV can be determined.

FIG. 11C shows the two lines (AD and CF) and two circles (with arcs DEFand ABC) that describe the shape of the FOV in the ultrasound image. Thetwo lines and circles can be used calculate the position of any pixel inthe image with respect to these lines and circles (i.e. to the left orright of a line, inside or outside of a circle). As shown in FIG. 11C,any pixel that is to the right of the left field line, to the left ofthe right field line, outside the circle defined by the upper curve andinside the circle defined by the lower curve is set to 1; all otherpixels are set to 0.

Ultrasound Overlay Visual Effects

According to various embodiments, visual effect can be applied to theultrasound image to be overlaid over the 3D image. Some examples ofvisual effects can include colorization and alpha blending. These visualeffects may be applied using fragment shaders. To apply the colorizationand alpha blending effects, the ultrasound system may determine theintensity of a pixel, and map the intensity to the colors red, green andblue. The mapping to each color can be weighed to determine how much ofthe intensity is to be applied to each color. In one example, theintensity is mapped to the color blue and the intensity is scaled so asto appear clearly overlaid over the 3D image.

Example Computer System

The embodiments of the systems and methods described herein may beimplemented as hardware or software, or a combination of both executingin one or more computer systems. There are many examples of computersystems that are currently in use. In an embodiment these systems andmethods are implemented in computer programs executing on programmablecomputers each comprising at least one processor, a data storage system(including volatile and non-volatile memory and/or storage elements), atleast one input device, and at least one output device. For example andwithout limitation, the programmable computers may be a mainframecomputer, server, personal computer, laptop, personal data assistant, orcellular telephone.

Further, aspects may be located on a single computer system (aworkstation) or may be distributed among a plurality of computer systemsconnected to one or more communications networks. For example, variousaspects and functions may be distributed among one or more computersystems configured to provide a service to one or more client computers,or to perform an overall task as part of a distributed system.Additionally, aspects may be performed on a client-server or multi-tiersystem that includes components distributed among one or more serversystems that perform various functions. Consequently, examples are notlimited to executing on any particular system or group of systems.Further, aspects and functions may be implemented in software, hardwareor firmware, or any combination thereof. Thus, aspects and functions maybe implemented within methods, acts, systems, system elements andcomponents using a variety of hardware and software configurations, andexamples are not limited to any particular distributed architecture,network, or communication protocol.

A distributed computer system, in which various aspects and functionsmay be practiced, may include one more computer systems that exchange(i.e. send or receive) information. The computer systems may beinterconnected by, and may exchange data through, a communicationnetwork. The network may include any communication network through whichcomputer systems may exchange data. To exchange data using the network,the computer systems and the network may use various methods, protocolsand standards, including, among others, Fibre Channel, Token Ring,Ethernet, Wireless Ethernet, Bluetooth, IP, IPV6, TCP/IP, UDP, DTN,HTTP, FTP, SNMP, SMS, MMS, SS7, JSON, SOAP, CORBA, REST and WebServices. To ensure data transfer is secure, the computer systems maytransmit data via the network using a variety of security measuresincluding, for example, TLS, SSL or VPN. The distributed computer systemdescribed herein is not so limited and may include any number ofcomputer systems and computing devices, networked using any medium andcommunication protocol.

A distributed computer system, in which various aspects and functionsmay be practiced, may include a processor, a memory, a bus, an interfaceand data storage. The processor may perform a series of instructionsthat result in manipulated data. The processor may be a commerciallyavailable processor such as an Intel Xeon, Itanium, Core, Celeron,Pentium, AMD Opteron, Sun UltraSPARC, IBM Power5+, or IBM mainframechip, but may be any type of processor, multiprocessor or controller.The processor is connected to other system components, including one ormore memory devices, by the bus.

The memory may be used for storing programs and data during operation ofthe computer system. Thus, the memory may be a relatively highperformance, volatile, random access memory such as a dynamic randomaccess memory (DRAM) or static memory (SRAM). However, the memory mayinclude any device for storing data, such as a disk drive or othernon-volatile storage device. Various examples may organize the memoryinto particularized and, in some cases, unique structures to perform thefunctions disclosed herein and these data structures may be tailored tostore values for particular types of data.

Components of the computer system may be coupled by an interconnectionelement such as the bus. The bus may include one or more physicalbusses, for example, busses between components that are integratedwithin a same machine, but may include any communication couplingbetween system elements including specialized or standard computing bustechnologies such as IDE, SCSI, PCI and InfiniBand. Thus, the busenables communications, such as data and instructions, to be exchangedbetween system components of the computer system.

The computer system also includes one or more interface devices such asinput devices or modules, output devices or modules and combinationinput/output devices or modules. Interface devices may receive input orprovide output. More particularly, output devices may render informationfor external presentation. Input devices may accept information fromexternal sources. Examples of interface devices include keyboards, mousedevices, trackballs, microphones, touch screens, printing devices,display screens, speakers, network interface cards, etc. Interfacedevices allow the computer system to exchange information andcommunicate with external entities, such as users and other computersystems.

The data storage may include a computer readable and writeablenonvolatile (non-transitory) data storage medium in which instructionsare stored that define a program or other object that may be executed bythe processor. The data storage also may include information that isrecorded, on or in, the medium, and this information may be processed bythe processor during execution of the program. More specifically, theinformation may be stored in one or more data structures specificallyconfigured to conserve storage space or increase data exchangeperformance. The instructions may be persistently stored as encodedsignals, and the instructions may cause the processor to perform any ofthe functions described herein. The medium may, for example, be opticaldisk, magnetic disk or flash memory, among others. In operation, theprocessor or some other controller may cause data to be read from thenonvolatile recording medium into another memory, such as the memory,that allows for faster access to the information by the processor thandoes the storage medium included in the data storage. The memory may belocated in the data storage or in the memory, however, the processor maymanipulate the data within the memory, and then copy the data to thestorage medium associated with the data storage after processing iscompleted. A variety of components may manage data movement between thestorage medium and other memory elements and examples are not limited toparticular data management components. Further, examples are not limitedto a particular memory system or data storage system.

The various aspects and functions are not limited to being implementedon the computer system described above. Various aspects and functionsmay be practiced on one or more computers having a differentarchitectures or components than described above. For instance, thecomputer system may include specially programmed, special-purposehardware, such as an application-specific integrated circuit (ASIC)tailored to perform a particular operation disclosed herein.

The computer system may be a computer system including an operatingsystem that manages at least a portion of the hardware elements includedin the computer system. In some examples, a processor or controller,such as the processor, executes an operating system. Examples of aparticular operating system that may be executed include a Windows-basedoperating system, such as, Windows NT, Windows 2000 (Windows ME),Windows XP, Windows Vista or Windows 7 operating systems, available fromthe Microsoft Corporation, a MAC OS System X operating system availablefrom Apple Computer, one of many Linux-based operating systemdistributions, for example, the Enterprise Linux operating systemavailable from Red Hat Inc., or a UNIX operating systems available fromvarious sources. Many other operating systems may be used, and examplesare not limited to any particular operating system.

The processor and operating system together define a computer platformfor which application programs in high-level programming languages maybe written. These component applications may be executable,intermediate, bytecode or interpreted code which communicates over acommunication network, for example, the Internet, using a communicationprotocol, for example, TCP/IP. Similarly, aspects may be implementedusing an object-oriented programming language, such as .Net, SmallTalk,Java, C++, Ada, or C# (C-Sharp). Other object-oriented programminglanguages may also be used. Alternatively, functional, scripting, orlogical programming languages may be used.

Additionally, various aspects and functions may be implemented in anon-programmed environment, for example, documents created in HTML, XMLor other format that, when viewed in a window of a browser program,render aspects of a graphical-user interface or perform other functions.Further, various examples may be implemented as programmed ornon-programmed elements, or any combination thereof. For example, a webpage may be implemented using HTML while a data object called fromwithin the web page may be written in C++. Thus, the examples are notlimited to a specific programming language and any suitable programminglanguage could be used. Thus, functional components disclosed herein mayinclude a wide variety of elements, e.g. executable code, datastructures or objects, configured to perform the functions describedherein.

In some examples, the components disclosed herein may read parametersthat affect the functions performed by the components. These parametersmay be physically stored in any form of suitable memory includingvolatile memory (such as RAM) or nonvolatile memory (such as a magnetichard drive). In addition, the parameters may be logically stored in apropriety data structure (such as a database or file defined by a usermode application) or in a commonly shared data structure (such as anapplication registry that is defined by an operating system). Inaddition, some examples provide for both system and user interfaces thatallow external entities to modify the parameters and thereby configurethe behavior of the components.

Having described above several aspects of at least one embodiment, it isto be appreciated various alterations, modifications, and improvementswill readily occur to those skilled in the art. Such alterations,modifications, and improvements are intended to be part of thisdisclosure and are intended to be within the scope of the invention.Accordingly, the foregoing description and drawings are by way ofexample only.

What is claimed is:
 1. A method overlaying a three-dimensional image ofa tissue and an image of the tissue from an imaging transducer, having afield of view, the method comprising: co-registering a first co-ordinatespace of the three-dimensional image with a second co-ordinate space ofthe field of view of the image from the imaging transducer; determiningpixel dimensions of the field of view in the second co-ordinate space;scaling the three-dimensional image in the first co-ordinate space tomatch the pixel dimensions of the field of view in the secondco-ordinate space; transforming position and orientation of thethree-dimensional image of the tissue in the first co-ordinate space tomatch position and orientation of the field of view in the secondco-ordinate space; and displaying an overlaid image comprising thethree-dimensional image and the field of view of the imaging transducer.2. The method of claim 1, wherein determining pixel dimensions of thefield of view further comprises: determining pixel coordinates of pointsof interest on an upper boundary of the field of view; and determiningpixel width and height based on geometry of the upper boundary of thefield of view.
 3. The method of claim 2, wherein the imaging transducercomprises a linear transducer, the upper boundary comprises a linesegment, and the points of interest comprise ends points and a midpointof the line segment.
 4. The method of claim 3, wherein determining thepixel width and height based on geometry further comprises determiningthe pixel width and height based on geometry of the line segment andaspect ratio of the image.
 5. The method of claim 2, wherein the imagingtransducer comprises a curvilinear transducer, the upper boundarycomprises an arc segment, and the points of interest comprise a midpointand a radius of curvature.
 6. The method of claim 5, wherein determiningthe pixel width and height based on geometry further comprisesdetermining the pixel width and height based on geometry of the arcsegment and aspect ratio of the image.
 7. The method of claim 1, furthercomprising detecting changes in image depth of the field of view of theimage and rescaling the field of view based on a scaling factor to matchan adjusted image depth of the field of view.
 8. The method of claim 1,further comprising: receiving the three-dimensional image of the tissuefrom a first imaging system; and receiving the image of the tissue fromthe imaging transducer from a second imaging system, wherein the firstimaging system is different from the second imaging system.
 9. Anapparatus for overlaying a three-dimensional image of a tissue and animage of the tissue from an imaging transducer, having a field of view,the apparatus comprising: a non-transitory computer readable mediumconfigured to store any of the three-dimensional image and the imagefrom the imaging transducer; a processor, coupled to the computerreadable medium and configured to: co-register a first co-ordinate spaceof the three-dimensional image with a second co-ordinate space of thefield of view of the image from the imaging probe; determine pixeldimensions of the field of view in the second co-ordinate space; scalethe three-dimensional image in the first co-ordinate space to match thepixel dimensions of the field of view in the second co-ordinate space;and transform position and orientation of the three-dimensional image inthe first co-ordinate space to match the position and orientation of thefield of view in the second co-ordinate space; and a display configuredto display an overlaid image comprising the three-dimensional image andthe field of view of the imaging transducer.
 10. The apparatus of claim9, wherein the processor is configured to: determine pixel coordinatesof points of interest on an upper boundary of the field of view; anddetermine pixel width and height based on geometry of the of the upperboundary of the field of view.
 11. The apparatus of claim 10, whereinthe imaging transducer comprises a linear transducer, the upper boundarycomprises a line segment, and the points of interest comprise endspoints and a midpoint of the line segment.
 12. The apparatus of claim10, wherein the imaging transducer comprises a curvilinear transducer,the upper boundary comprises an arc segment, and the points of interestcomprise a midpoint and a radius of curvature.
 13. The apparatus ofclaim 9, wherein the processor is further configured to detect changesin image depth of the field of view of the image and rescale the fieldof view based on a scaling factor to match an adjusted image depth ofthe field of view.
 14. The apparatus of claim 9, further comprising aninput module configured to receive the three-dimensional image of thetissue from a first imaging system and configured to receive the imageof the tissue from the imaging transducer from a second imaging system,wherein the first imaging system is different from the second imagingsystem.
 15. A method overlaying an image of a tissue from an imagingsystem, having an imaging transducer with a field of view, and athree-dimensional image of the tissue, the method comprising:co-registering a first co-ordinate space of the three-dimensional imagewith a second co-ordinate space of the field of view of the image fromthe imaging transducer; receiving input bounds of the field of view anddepth markers identified in the image; determining pixel spacing basedon depth associated with the depth markers in the image; and scaling thethree-dimensional image in the first co-ordinate space to match thepixel spacing of the field of view in the second co-ordinate space;transforming position and orientation of the three-dimensional image ofthe tissue in the first co-ordinate space to match the position andorientation of the field of view in the second co-ordinate space; anddisplaying an overlaid image comprising the field of view of the imagingtransducer and the three-dimensional image.
 16. The method of claim 15,further comprising receiving depth associated with each of the depthmarkers in the field of view.
 17. The method of claim 15, furthercomprising receiving input identifying a freeze state of the image fromthe transducer.
 18. The method of claim 15, further comprisingdetermining co-ordinates of points of interest defining center, upperand lower bounds of the field of view and determining pixel spacingbased on the co-ordinates of the point of interest.
 19. The method ofclaim 15, further comprising determining whether the field of view ofthe image is cropped.
 20. The method of claim 15, wherein the imagingtransducer comprises a curvilinear transducer, and the method furthercomprises detecting an angle of the field of view of the curvilineartransducer.
 21. The method of claim 15, further comprising determining aset of pixels configured to define the field of view and masking outanother set of pixels outside the defined the set of pixels defining thefield of view.
 22. The method of claim 15, further comprising: receivingthe three-dimensional image of the tissue from a first imaging system;and receiving the image of the tissue from the imaging transducer from asecond imaging system, wherein the first imaging system is differentfrom the second imaging system.
 23. An apparatus for overlaying an imageof a tissue from an imaging system, having an imaging transducer with afield of view, and a three-dimensional image of the tissue, theapparatus comprising: a non-transitory computer readable mediumconfigured to store any of the three-dimensional image and the imagefrom the imaging transducer; a processor coupled to the computerreadable medium and configured to: co-register a first co-ordinate spaceof the three-dimensional image with a second co-ordinate space of thefield of view of the image from the imaging transducer; receive inputidentifying bounds of the field of view and depth markers in the image;determine pixel spacing based on depth associated with the depth markersin the image; and scale the three-dimensional image in the firstco-ordinate space to match the pixel spacing of the field of view in thesecond co-ordinate space; and transform position and orientation of thethree-dimensional image in the first co-ordinate space to match theposition and orientation of the field of view in the second co-ordinatespace; and a display configured to display an overlaid image comprisingthe field of view of the imaging transducer and the three-dimensionalimage.
 24. The apparatus of claim 23, further comprising an input moduleconfigured to receive the three-dimensional image of the tissue from afirst imaging system and configured to receive the image of the tissuefrom the imaging transducer from a second imaging system, wherein thefirst imaging system is different from the second imaging system.
 25. Amethod of calibrating an imaging system for overlaying an image of atissue from the imaging system and a three-dimensional image of thetissue, the imaging system having an imaging transducer with a field ofview, the method comprising: providing for a user to select bounds ofthe field of view of the imaging transducer; providing for the user toselect depth marker indicators in the image; cycling through the depthmarker indicators to determine a specific depth marker for each depth;and calculating coordinates of points of interest in the field of viewof the image; and calculating pixel spacing based on the coordinates ofthe points of interest and depth associated with the depth markers inthe image.
 26. The method of claim 25, further comprising providing forthe user to select an area identifying a freeze state of the image fromthe transducer.
 27. The method of claim 25, further comprising providingfor the user to select a background threshold, wherein the backgroundthreshold is used to filter an ultrasound signal overlapping with thedepth marker indicators.
 28. The method of claim 25, further comprisingdetermining whether the field of view of the image is cropped.
 29. Themethod of claim 25, wherein the imaging transducer comprises acurvilinear transducer, and the method further comprises detecting anangle of the field of view of the curvilinear transducer.